{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzWu-60xBDwe"
      },
      "source": [
        "Copyright 2024 Google LLC."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "fXkE8WP5BIer"
      },
      "outputs": [],
      "source": [
        "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KYCPCcxctDvx"
      },
      "source": [
        "# Fine-tune PaliGemma with Keras\n",
        "\n",
        "This notebook shows how to fine-tune [PaliGemma](https://ai.google.dev/gemma/docs/paligemma) on a vision-language task with [Keras](https://keras.io/). *Fine-tuning* is a process that can improve your model's performance on specific tasks or help the model adhere to specific output requirements when instructions aren't sufficient and you have a set of examples that demonstrate the outputs you want. Gemma-based models like PaliGemma require fine-tuning to produce expected results.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/PaliGemma/[PaliGemma_1]Finetune_with_Keras.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FvoBtrn4uvLg"
      },
      "source": [
        "## Before you begin\n",
        "\n",
        "Before going through this notebook, you should be familiar with Python code, as well as how large language models (LLMs) are trained. You don't need to be familiar with Keras, but basic knowledge about Keras (or similar technologies) is helpful when reading through the example code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OmoxnqUAxJzg"
      },
      "source": [
        "## Setup\n",
        "\n",
        "The following sections explain the preliminary steps for getting a notebook to use a PaliGemma model, including model access, getting an API key, and configuring the notebook runtime."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w1UH8Qjhxe5a"
      },
      "source": [
        "### Get access to PaliGemma\n",
        "\n",
        "Before using PaliGemma for the first time, you must request access to the model through Kaggle by completing the following steps:\n",
        "\n",
        "1. Log in to [Kaggle](https://www.kaggle.com/), or create a new Kaggle account if you don't already have one.\n",
        "2. Go to the [PaliGemma model card](https://www.kaggle.com/models/google/paligemma/) and click **Request Access**.\n",
        "3. Complete the consent form and accept the terms and conditions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "82Et2NA6zuxg"
      },
      "source": [
        "### Configure your API key\n",
        "\n",
        "To use PaliGemma, you must provide your Kaggle username and a Kaggle API key.\n",
        "\n",
        "To generate a Kaggle API key, open your [**Settings** page in Kaggle](https://www.kaggle.com/settings) and click **Create New Token**. This triggers the download of a `kaggle.json` file containing your API credentials.\n",
        "\n",
        "Then, in Colab, select **Secrets** (🔑) in the left pane and add your Kaggle username and Kaggle API key. Store your username under the name `KAGGLE_USERNAME` and your API key under the name `KAGGLE_KEY`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CV8xgN_rzyGR"
      },
      "source": [
        "### Select the runtime\n",
        "\n",
        "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the PaliGemma model. In this case, you can use a T4 GPU:\n",
        "\n",
        "1. In the upper-right of the Colab window, click the **▾ (Additional connection options)** dropdown menu.\n",
        "1. Select **Change runtime type**.\n",
        "1. Under **Hardware accelerator**, select **T4 GPU**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pUp359rD0bTe"
      },
      "source": [
        "### Set environment variables\n",
        "\n",
        "Set the environment variables for `KAGGLE_USERNAME` and `KAGGLE_KEY`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zGLIp1Cx3_CX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata\n",
        "\n",
        "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n",
        "# vars as appropriate or make your credentials available in ~/.kaggle/kaggle.json\n",
        "\n",
        "os.environ[\"KAGGLE_USERNAME\"] = userdata.get('KAGGLE_USERNAME')\n",
        "os.environ[\"KAGGLE_KEY\"] = userdata.get('KAGGLE_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCd__uzW_eK-"
      },
      "source": [
        "### Install KerasNLP"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DfxKb3F839Ks"
      },
      "outputs": [],
      "source": [
        "!pip install -U -q keras-nlp"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZOxk5hUaX5la"
      },
      "outputs": [],
      "source": [
        "# Set the backbend before importing Keras\n",
        "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n",
        "# Avoid memory fragmentation on JAX backend.\n",
        "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QlCp6b2VqNfy"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 1\n",
        "TRAIN_EXAMPLES = 4\n",
        "LEARNING_RATE = 0.003\n",
        "\n",
        "TRAIN_STEPS = TRAIN_EXAMPLES // BATCH_SIZE\n",
        "EVAL_STEPS = TRAIN_STEPS // 4"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zDoq0O77GF30"
      },
      "source": [
        "## Download the training and validation data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AZeUDK3YX6Zr"
      },
      "outputs": [],
      "source": [
        "import io\n",
        "import json\n",
        "import os\n",
        "import urllib\n",
        "\n",
        "import base64\n",
        "import html\n",
        "\n",
        "import numpy as np\n",
        "import keras\n",
        "import keras_nlp\n",
        "import tensorflow as tf\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from IPython.core.display import display, HTML\n",
        "from PIL import Image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dTfe2k8J4Bw0"
      },
      "outputs": [],
      "source": [
        "train_file = urllib.request.urlopen(\n",
        "    \"https://storage.googleapis.com/longcap100/data_train90.jsonl\"\n",
        ")\n",
        "val_file = urllib.request.urlopen(\n",
        "    \"https://storage.googleapis.com/longcap100/data_val10.jsonl\"\n",
        ")\n",
        "\n",
        "# Crop the image to the desired dimensions.\n",
        "target_size = (224, 224)\n",
        "\n",
        "def load_image(image_url):\n",
        "    image = tf.io.decode_jpeg(urllib.request.urlopen(image_url).read())\n",
        "    return tf.image.resize(image, size=target_size)\n",
        "\n",
        "def load_dataset(file):\n",
        "    captions = []\n",
        "    images = []\n",
        "    for line in file:\n",
        "        sample = json.loads(line)\n",
        "        captions.append(sample[\"suffix\"])\n",
        "        image_name = sample[\"image\"]\n",
        "        image_url = f\"https://storage.googleapis.com/longcap100/{image_name}\"\n",
        "        images.append(load_image(image_url))\n",
        "    return tf.data.Dataset.from_tensor_slices({\n",
        "        \"images\": images,\n",
        "        \"prompts\": [\"caption en\\n\"] * len(images),\n",
        "        \"responses\": captions,\n",
        "    })\n",
        "\n",
        "train_data = load_dataset(train_file).shuffle(1000).batch(BATCH_SIZE)\n",
        "val_data = load_dataset(val_file).shuffle(1000).batch(BATCH_SIZE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J43VQ92hgJcd"
      },
      "source": [
        "## View training examples\n",
        "\n",
        "In this notebook, the training data contains 90 images that are paired with long descriptions of what's depicted in the image.\n",
        "\n",
        "**Note:** Normal training data sets that are meant to be used for practical use cases should contain more images, but this notebook limits the number of data points so that you can train the model in a reasonable amount of time for an example.\n",
        "\n",
        "The code below prints a random selection of images with their descriptions from the training data set so that you can see what the images and descriptions your model is trained on looks like. Each image is displayed in as a 128x128 pixel JPEG, with the description printed next to the image to the right."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IPKB8yK8gw8g"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training examples\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A person holds a yellow popsicle in their hand, the popsicle adorned with text psecre on the stick. The person wears a green jacket with a red collar, a white shirt, and silver rings on their finger. The jacket has a zipper on the front. </p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A tool box filled with a variety of tools, including a wrench with a silver head, a screwdriver with a gray handle,a wrench with a gray head, a screwdriver with a gray handle, a metal socket with a silver head...</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A yellow train is stopped at a station. The train is on the tracks, and the lights are on. The platform has a white line and a white stripe on the platform. There is a person standing on the platform, and a person walking on the platform. The train has a number on its side. The train is moving, and the doors are closed.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">Two champagne glasses sit on a ledge overlooking the ocean, reflecting the setting sun. The glasses are filled with bubbly champagne, and the bubbles dance in the liquid. The sky is clear, and the water is calm. The mountains loom large in the distance, and the rocky cliffs on the shore line the water&#x27;s edge. </p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A red train travels down a snow-covered street, its headlights illuminating the snow. The train has a red and white Coca-Cola logo on its side, and the snow is piled up on the side of the road. A green traffic light is on the side of the road. The train has a number on its front, and the snow is white. The street light is green, and the traffic light is also green. The train has a windshield wiper, and the snow is on the ground.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A room with a ping pong table and a desk. The room has a large window and a door. There is a ping pong table with a blue top in the middle and black lamps on the right. The floor is wooden. There is a black chair on the left and four desk chairs on the right. The room has a lot of lights hanging from the ceiling. The room also has a lot of decorations on the wall.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">Cinque Terre city perched on the side of a towering cliff, overlooking the serene ocean. The sky above is adorned with fluffy white clouds, while the water below mirrors the sky in its tranquility. The buildings below are painted in vibrant hues, ranging from yellow to orange to pink. The hillside is adorned with lush greenery, and the rocky shore provides a picturesque backdrop. The overall atmosphere is serene and captivating, offering a glimpse into the heart of this idyllic town.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">A white tray with a green toothbrush and a metal dental mirror sits on a white wall. The tray is square and has a white paper on it. The toothbrush is green. The mirror is on a metal tray and has a white paper on it. There is also a small green object on the tray. The wall is white and the window is black.</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def render_inline(image, resize=(224, 224)):\n",
        "  \"\"\"Convert image into inline html.\"\"\"\n",
        "  image = tf.keras.preprocessing.image.array_to_img(image)\n",
        "  image.resize(resize)\n",
        "  with io.BytesIO() as buffer:\n",
        "    image.save(buffer, format='jpeg')\n",
        "    image_b64 = str(base64.b64encode(buffer.getvalue()), \"utf-8\")\n",
        "    return f\"data:image/jpeg;base64,{image_b64}\"\n",
        "\n",
        "def render_example(image, caption):\n",
        "  image = np.asarray(image)\n",
        "  return f\"\"\"\n",
        "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
        "        <img style=\"width:128px; height:128px;\" src=\"{render_inline(image, resize=(64,64))}\" />\n",
        "        <p style=\"width:256px; margin:10px; font-size:small;\">{html.escape(caption)}</p>\n",
        "    </div>\n",
        "    \"\"\"\n",
        "\n",
        "html_out = \"\"\n",
        "\n",
        "for element in train_data.take(8):\n",
        "  caption = tf.compat.as_str_any(element[\"responses\"].numpy()[0])\n",
        "  html_out += render_example(element[\"images\"].numpy()[0], caption)\n",
        "\n",
        "print(\"Training examples\")\n",
        "display(HTML(html_out))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qa6cn8nXJx51"
      },
      "source": [
        "## Load Model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4eoMtlGCaEPg"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"pali_gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"pali_gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaTokenizer</span>)          │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">257,152</span> │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (\u001b[38;5;33mPaliGemmaTokenizer\u001b[0m)          │                                             \u001b[38;5;34m257,152\u001b[0m │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"pali_gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"pali_gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>)       │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,923,335,408</span> │ images[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],              │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaBackbone</span>)           │                           │                 │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│                               │                           │                 │ response_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
              "│                               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">526,647,296</span> │ pali_gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]  │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GetItem</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ token_embedding[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (\u001b[38;5;33mInputLayer\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m3\u001b[0m)       │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (\u001b[38;5;33mInputLayer\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,923,335,408\u001b[0m │ images[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],              │\n",
              "│ (\u001b[38;5;33mPaliGemmaBackbone\u001b[0m)           │                           │                 │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│                               │                           │                 │ response_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
              "│                               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │     \u001b[38;5;34m526,647,296\u001b[0m │ pali_gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]  │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (\u001b[38;5;33mGetItem\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │               \u001b[38;5;34m0\u001b[0m │ token_embedding[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,923,335,408</span> (10.89 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,923,335,408\u001b[0m (10.89 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,923,335,408</span> (10.89 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,923,335,408\u001b[0m (10.89 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# use half-precision to save memory\n",
        "#keras.config.set_floatx('bfloat16') # BUG?\n",
        "\n",
        "pali_gemma_lm = keras_nlp.models.PaliGemmaCausalLM.from_preset(\n",
        "    \"pali_gemma_3b_224\"\n",
        ")\n",
        "pali_gemma_lm.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAx6m6o8aTy7"
      },
      "source": [
        "## Inference before fine tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gjdx0Ocxaap7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "cow standing on a beach</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Inference Result\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "a red blazer and black bag , a fashion item of the week</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "person in a long shot of our model</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "white hangers on a rack in the bedroom</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "this graphic sweatshirt is a must have for your wardrobe .</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "test_image_url = 'https://storage.googleapis.com/keras-cv/models/paligemma/cow_beach_1.png'\n",
        "test_image = load_image(test_image_url)\n",
        "\n",
        "def inference_test(image):\n",
        "  prompt = 'caption en\\n'\n",
        "  output = pali_gemma_lm.generate(\n",
        "      inputs={\n",
        "          \"images\": image,\n",
        "          \"prompts\": prompt,\n",
        "      }\n",
        "  )\n",
        "  return render_example(image, output)\n",
        "\n",
        "display(HTML(inference_test(test_image)))\n",
        "\n",
        "\n",
        "def make_predictions():\n",
        "  html_out = \"\"\n",
        "  for element in val_data.take(4):\n",
        "    html_out += inference_test(element[\"images\"].numpy()[0])\n",
        "\n",
        "  print(\"\\nInference Result\")\n",
        "  display(HTML(html_out))\n",
        "\n",
        "make_predictions()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJbdY9ehahEK"
      },
      "source": [
        "## LoRA Fine-tuning"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VvgDhGKhIHvG"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Preprocessor: \"pali_gemma_causal_lm_preprocessor\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mPreprocessor: \"pali_gemma_causal_lm_preprocessor\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Tokenizer (type)                                   </span>┃<span style=\"font-weight: bold\">                                             Vocab # </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaTokenizer</span>)          │                                             <span style=\"color: #00af00; text-decoration-color: #00af00\">257,152</span> │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mTokenizer (type)                                  \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m                                            Vocab #\u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ pali_gemma_tokenizer (\u001b[38;5;33mPaliGemmaTokenizer\u001b[0m)          │                                             \u001b[38;5;34m257,152\u001b[0m │\n",
              "└────────────────────────────────────────────────────┴─────────────────────────────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"pali_gemma_causal_lm\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"pali_gemma_causal_lm\"\u001b[0m\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)                  </span>┃<span style=\"font-weight: bold\"> Output Shape              </span>┃<span style=\"font-weight: bold\">         Param # </span>┃<span style=\"font-weight: bold\"> Connected to               </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">224</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">3</span>)       │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)     │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)              │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">2048</span>)        │   <span style=\"color: #00af00; text-decoration-color: #00af00\">2,924,699,376</span> │ images[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],              │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PaliGemmaBackbone</span>)           │                           │                 │ padding_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],        │\n",
              "│                               │                           │                 │ response_mask[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],       │\n",
              "│                               │                           │                 │ token_ids[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │     <span style=\"color: #00af00; text-decoration-color: #00af00\">526,647,296</span> │ pali_gemma_backbone[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]  │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">ReversibleEmbedding</span>)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GetItem</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">257152</span>)      │               <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ token_embedding[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                 \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape             \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m        Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to              \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
              "│ images (\u001b[38;5;33mInputLayer\u001b[0m)           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m224\u001b[0m, \u001b[38;5;34m3\u001b[0m)       │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m)     │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ response_mask (\u001b[38;5;33mInputLayer\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_ids (\u001b[38;5;33mInputLayer\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)              │               \u001b[38;5;34m0\u001b[0m │ -                          │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ pali_gemma_backbone           │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2048\u001b[0m)        │   \u001b[38;5;34m2,924,699,376\u001b[0m │ images[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],              │\n",
              "│ (\u001b[38;5;33mPaliGemmaBackbone\u001b[0m)           │                           │                 │ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],        │\n",
              "│                               │                           │                 │ response_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],       │\n",
              "│                               │                           │                 │ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ token_embedding               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │     \u001b[38;5;34m526,647,296\u001b[0m │ pali_gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m]  │\n",
              "│ (\u001b[38;5;33mReversibleEmbedding\u001b[0m)         │                           │                 │                            │\n",
              "├───────────────────────────────┼───────────────────────────┼─────────────────┼────────────────────────────┤\n",
              "│ get_item (\u001b[38;5;33mGetItem\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m257152\u001b[0m)      │               \u001b[38;5;34m0\u001b[0m │ token_embedding[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m0\u001b[0m]      │\n",
              "└───────────────────────────────┴───────────────────────────┴─────────────────┴────────────────────────────┘\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,924,699,376</span> (10.90 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,924,699,376\u001b[0m (10.90 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,363,968</span> (5.20 MB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m1,363,968\u001b[0m (5.20 MB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">2,923,335,408</span> (10.89 GB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,923,335,408\u001b[0m (10.89 GB)\n"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Enable lora to freeze most of the model and save memory.\n",
        "pali_gemma_lm.backbone.enable_lora(4)\n",
        "pali_gemma_lm.summary()\n",
        "\n",
        "# Lower our sequence length to further save memory.\n",
        "pali_gemma_lm.preprocessor.sequence_length = 64\n",
        "\n",
        "# Use Cosine Decay Scheduler with Warm up\n",
        "#cosine_decay_scheduler = tf.keras.optimizers.schedules.CosineDecay(\n",
        "#    initial_learning_rate = 0,\n",
        "#    decay_steps = TRAIN_EXAMPLES,\n",
        "#    warmup_target = LEARNING_RATE,\n",
        "#    warmup_steps = TRAIN_EXAMPLES / 10\n",
        "#)\n",
        "\n",
        "def plot_scheduler(step, scheduler):\n",
        "  x = range(step)\n",
        "  y = []\n",
        "  for step in x:\n",
        "    y.append(scheduler(step))\n",
        "  plt.plot(x, y, label=scheduler.name)\n",
        "  plt.xlabel('Epoch')\n",
        "  plt.ylabel('Learning Rate')\n",
        "  plt.legend()\n",
        "  plt.show()\n",
        "\n",
        "#plot_scheduler(TRAIN_EXAMPLES, cosine_decay_scheduler)\n",
        "\n",
        "# Use AdamW (a common optimizer for transformer models).\n",
        "#optimizer = keras.optimizers.SGD(learning_rate=cosine_decay_scheduler)\n",
        "optimizer = keras.optimizers.AdamW(learning_rate=LEARNING_RATE)\n",
        "\n",
        "pali_gemma_lm.compile(\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    optimizer=optimizer,\n",
        "    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oVTG6rXX9oF2"
      },
      "source": [
        "## Fine-tune the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h8FW7mMGeY3S"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m204s\u001b[0m 2s/step - loss: 1.6644 - sparse_categorical_accuracy: 0.5384\n",
            "Epoch 2/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m115s\u001b[0m 1s/step - loss: 0.9424 - sparse_categorical_accuracy: 0.6918\n",
            "Epoch 3/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m143s\u001b[0m 2s/step - loss: 0.6737 - sparse_categorical_accuracy: 0.7758\n",
            "Epoch 4/4\n",
            "\u001b[1m90/90\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m143s\u001b[0m 2s/step - loss: 0.5461 - sparse_categorical_accuracy: 0.8162\n"
          ]
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFPklEQVR4nO3deVxU9eI+8OfMwAz7KLIJDO77AsgWmpqluV3SNlFLULOy1Fzur5Kbacu3uN0WtaLUUmlTXHKpNJcsxS3ZXcENFFRWlRlAGWDm/P7wRnETZRDmMDPP+/U6f3j8HM8z5zXNPJ0553wEURRFEBEREUlEJnUAIiIism4sI0RERCQplhEiIiKSFMsIERERSYplhIiIiCTFMkJERESSYhkhIiIiSbGMEBERkaRspA7QEAaDAVeuXIGzszMEQZA6DhERETWAKIooKyuDt7c3ZLL6z3+YRRm5cuUK1Gq11DGIiIioEfLy8uDr61vv35tFGXF2dgZw68W4uLhInIaIiIgaQqvVQq1W136P18csysgfP824uLiwjBAREZmZu11iwQtYiYiISFIsI0RERCQplhEiIiKSFMsIERERSYplhIiIiCTFMkJERESSYhkhIiIiSRldRhITExEREQFvb28IgoAtW7Y0eNuDBw/CxsYGAQEBxu6WiIiILJTRZaSiogL+/v6Ii4szarvS0lJERUXhoYceMnaXREREZMGMfgLryJEjMXLkSKN3NH36dEycOBFyudyosylERERk2Uxyzcjq1auRnZ2NRYsWNWi8TqeDVqutsxAREZFlavYycvbsWcyfPx/ffvstbGwadiImNjYWKpWqduGMvURERJarWcuIXq/HxIkT8eabb6Jr164N3i4mJgYajaZ2ycvLa5Z8p65oMWHF77hWUdUs/z4RERHdXbPO2ltWVoaUlBSkp6dj5syZAACDwQBRFGFjY4Ndu3bhwQcf/Nt2SqUSSqWyOaPBYBAxd10GTheW4flvUvDttDAobeTNuk8iIiL6u2Y9M+Li4oLjx48jIyOjdpk+fTq6deuGjIwMhIWFNefu70gmE/DJxEA4K22QfOE6Xtl4DKIoSpaHiIjIWhl9ZqS8vBznzp2r/XNOTg4yMjLg6uoKPz8/xMTE4PLly/j6668hk8nQu3fvOtt7eHjAzs7ub+ul0NXTGZ8/HYTJq5OwNeMK2rk6YN7D3aSORUREZFWMPjOSkpKCwMBABAYGAgDmzZuHwMBALFy4EACQn5+P3Nzcpk3ZjO7v4ob/G3urGH386zlsTL0kcSIiIiLrIohm8NuEVquFSqWCRqOBi4tLs+zjvR1Z+HzvedjKBXw1NRT9O7k1y36IiIisRUO/vzk3zX+9/HA3jO7bFtV6EdO/ScW5onKpIxEREVkFlpH/kskEfPikP/r5tYK2sgZT4pNwtVwndSwiIiKLxzLyF3a2cnwRFQw/VwfkXbuJaV+noLJaL3UsIiIii8Yy8j/aOCmxanIIXOxskJ5bin+uPwqDocVfVkNERGS2WEZuo7OHE5ZPCoatXMC24/l4f9dpqSMRERFZLJaReoR3aoN/P9YXAPD53vNISDKf25WJiIjMCcvIHTwe5IuXHuoCAHhtywnsP1sscSIiIiLLwzJyF3OHdsGYAG/oDSJe/DYNpwvKpI5ERERkUVhG7kIQBPznib4Ibe+KMl0NpsYno6isUupYREREFoNlpAGUNnIsnxSEDm6OuFx6E9O+SsHNKt7yS0RE1BRYRhqotaMCqyeHoLWDLY5d0mDOunToecsvERHRPWMZMUJ7N0esiAqGQi7DzpOF+PfPmVJHIiIiMnssI0YKae+K95+8dcvvF/tz8M3vFyVOREREZN5YRhphTIAP/jmsKwBg0dYT+C2rSOJERERE5otlpJFmPtgZTwT5wiACM9ek4dQVrdSRiIiIzBLLSCMJgoB3H+2D8I5tUFGlx9T4ZBRoeMsvERGRsVhG7oHCRoZlTwehk7sjCrSVeOarZFToaqSORUREZFZYRu6RysEWqyeHoo2jAievaPHSWt7yS0REZAyWkSbg18YBX0QHQ2kjw56sIrz90ympIxEREZkNlpEm0s+vNRZHBgAA4g9dwOqDOdIGIiIiMhMsI01oVJ+2mD+yOwDgrZ9OYfepQokTERERtXwsI03s+UEdMSFUDVEEXlqbjuOXNFJHIiIiatFYRpqYIAh4a0xvDOzihpvVejzzVTKulN6UOhYREVGLxTLSDGzlMsQ91Q/dPJ1RVKbD1PhklFVWSx2LiIioRWIZaSYudrZYNSUE7s5KZBWUYcaadNToDVLHIiIianFYRpqRTyt7rIwOhp2tDIlnirHwh5MQRT6DhIiI6K9YRppZX99WWDo+EIIArDmSiy/385ZfIiKiv2IZMYHhvbzw2qgeAIB3f87Ez8fzJU5ERETUcrCMmMgz93fApPvaQRSBOesykJ57XepIRERELQLLiIkIgoBFET0xpJs7dDUGPPt1CvKu3ZA6FhERkeSMLiOJiYmIiIiAt7c3BEHAli1b7jj+wIEDGDBgANq0aQN7e3t0794dixcvbmxes2Yjl+GTif3Qo60LSsqrMDU+GZqbvOWXiIism9FlpKKiAv7+/oiLi2vQeEdHR8ycOROJiYnIzMzEggULsGDBAqxYscLosJbASWmDVZOD4emixNmicrz4XSqqecsvERFZMUG8h3tNBUHA5s2bMXbsWKO2e+yxx+Do6IhvvvmmQeO1Wi1UKhU0Gg1cXFwakbTlOXFZg3HLD+NGlR7jgn3x3uN9IQiC1LGIiIiaTEO/v01+zUh6ejoOHTqEwYMH1ztGp9NBq9XWWSxNbx8VPp0YCJkArE+5hM/2npc6EhERkSRMVkZ8fX2hVCoRHByMGTNmYNq0afWOjY2NhUqlql3UarWpYprUg9098cYjvQAA7+88jR+PXpE4ERERkemZrIzs378fKSkpWLZsGZYsWYK1a9fWOzYmJgYajaZ2ycvLM1VMk4sKb4+pAzoAAP654ShSL16TOBEREZFp2ZhqRx063PrC7dOnDwoLC/HGG29gwoQJtx2rVCqhVCpNFU1yr43ugdxrN/BLZiGe/ToVm1/sj3ZtHKWORUREZBKSPGfEYDBAp9NJsesWSS4T8PGEAPT2ccG1iipMiU9G6Y0qqWMRERGZhNFlpLy8HBkZGcjIyAAA5OTkICMjA7m5uQBu/cQSFRVVOz4uLg4//vgjzp49i7Nnz2LlypX44IMP8PTTTzfNK7AQDgobrIoOgbfKDtnFFXjum1ToavRSxyIiImp2Rv9Mk5KSgiFDhtT+ed68eQCA6OhoxMfHIz8/v7aYALfOgsTExCAnJwc2Njbo1KkT3nvvPTz//PNNEN+yeLjYYdWUEDzx+WEk5VxDzPfH8eE4f97yS0REFu2enjNiKpb4nJE72XemGFPjk6E3iJg7tCtmD+0idSQiIiKjtdjnjNDdDe7qjrfH9AYALP7lDDanX5I4ERERUfNhGWmhJob54flBHQEAr248jiPZVyVORERE1DxYRlqwV0d0x8jeXqjSG/DcN6k4X1wudSQiIqImxzLSgslkAhZHBiBA3Qqam9WYGp+MaxW85ZeIiCwLy0gLZ2crxxdRwfBtbY+LV2/gua9TUFnNW36JiMhysIyYAXdnJVZPDoGznQ1SLl7HyxuPwWBo8TdBERERNQjLiJno4umMZU8HwUYm4MejV/DR7jNSRyIiImoSLCNmZEBnN7z7WB8AwKe/ncP6FMudQJCIiKwHy4iZGResxswhnQEA/9p0HIfOlUiciIiI6N6wjJihecO6IsLfGzUGEc9/m4pzRWVSRyIiImo0lhEzJJMJeP+Jvghq1xpllTWYvDoZxWWcBZmIiMwTy4iZ+uOW33ZtHHDp+k08y1t+iYjITLGMmDFXRwVWTw6Byt4WGXmlmLsug7f8EhGR2WEZMXMd3Z2wYlIQbOUCfj5RgPd2ZkkdiYiIyCgsIxYgrGMb/OeJvgCA5fuyseZIrsSJiIiIGo5lxEI8GuiLOUO7AABe33oCiWeKJU5ERETUMCwjFmT2Q13wWKAP9AYRL36XhqwCrdSRiIiI7oplxIIIgoDYx/sgrIMrynU1mLo6GUXaSqljERER3RHLiIVR2sixfFIQOro54oqmEs98lYIbVTVSxyIiIqoXy4gFauWgwOopIXB1VOD4ZQ1mJ2RAz1t+iYiohWIZsVDt2jhixaQgKGxk2H2qEO9uz5Q6EhER0W2xjFiw4Pau+PBJfwDAygM5+PrwBWkDERER3QbLiIWL8PfGy8O7AQDe+OEkfs0qlDgRERFRXSwjVuDFBzphXLAvDCIwc006Tl7RSB2JiIioFsuIFRAEAe882gcDOrfBjSo9psYnI19zU+pYREREAFhGrIatXIbPngpCZw8nFGp1eCY+BeU63vJLRETSYxmxIip7W6yeHAI3JwVO5Wsxa00aavQGqWMREZGVYxmxMmpXB3wZHQKljQy/nS7Gmz+egijyGSRERCQdlhErFKBuhSWRARAE4JvfL2LVwQtSRyIiIivGMmKlRvZpi5iR3QEA/7ftFHadLJA4ERERWSujy0hiYiIiIiLg7e0NQRCwZcuWO47ftGkThg0bBnd3d7i4uCA8PBw7d+5sbF5qQs8O7IiJYX4QRWB2QgaOXSqVOhIREVkho8tIRUUF/P39ERcX16DxiYmJGDZsGLZv347U1FQMGTIEERERSE9PNzosNS1BEPDWI70wqKs7blbr8cxXKbh0/YbUsYiIyMoI4j1cvSgIAjZv3oyxY8catV2vXr0QGRmJhQsXNmi8VquFSqWCRqOBi4tLI5LSnZRVVuPJZYeRVVCGbp7O2PBCOFzsbKWORUREZq6h398mv2bEYDCgrKwMrq6u9Y7R6XTQarV1Fmo+zna2WDU5BB7OSpwuLMOM79JQzVt+iYjIRExeRj744AOUl5dj3Lhx9Y6JjY2FSqWqXdRqtQkTWifvVvZYGR0Ce1s59p8twcKtJ3jLLxERmYRJy8iaNWvw5ptvYv369fDw8Kh3XExMDDQaTe2Sl5dnwpTWq4+vCh9PCIQgAGuT8rAiMVvqSEREZAVMVkYSEhIwbdo0rF+/HkOHDr3jWKVSCRcXlzoLmcawnp54fXRPAEDsz1nYfjxf4kRERGTpTFJG1q5diylTpmDt2rUYPXq0KXZJ92Dq/R0wuX97AMDcdRlIy70ubSAiIrJoRpeR8vJyZGRkICMjAwCQk5ODjIwM5ObmArj1E0tUVFTt+DVr1iAqKgoffvghwsLCUFBQgIKCAmg0nMa+JXv9Hz3xUHcP6GoMeParFORd4y2/RETUPIwuIykpKQgMDERgYCAAYN68eQgMDKy9TTc/P7+2mADAihUrUFNTgxkzZqBt27a1y+zZs5voJVBzkMsEfDwhEL28XXC1ogqTVydBc6Na6lhERGSB7uk5I6bC54xIp0BTibFxB1GgrUT/Tm0QPyUUChvOIkBERHfXYp8zQubFS2WHVZND4KiQ49D5q/jX5uO85ZeIiJoUywjdVU9vF3z6VD/IBGBj6iXE/XZO6khERGRBWEaoQYZ088CbY3oDAD7YdQZbMy5LnIiIiCwFywg12KT72mHa/R0AAC9vOIbkC9ckTkRERJaAZYSMEjOqB4b38kSV3oDnvk7BhZIKqSMREZGZYxkho8hlApZEBsLfV4XrN6oxJT4Z1yuqpI5FRERmjGWEjGavkOOL6GD4tLJHTkkFnv8mFboavdSxiIjITLGMUKN4ON+65ddZaYOkC9fw6sZjvOWXiIgahWWEGq2blzM+e7of5DIBWzKuYMkvZ6WOREREZohlhO7JwC7ueGfsrVt+l+45i+9TL0mciIiIzA3LCN2z8aF+eOGBTgCA+ZuO4fD5qxInIiIic8IyQk3i5Ye7YXSftqjWi3j+mxScKyqXOhIREZkJlhFqEjKZgA/H+SPQrxW0lTWYGp+Mq+U6qWMREZEZYBmhJmNnK8cXUcFQu9oj99oNPPt1CiqrecsvERHdGcsINSk3JyVWTw6Bi50N0nJL8f82HIXBwFt+iYiofiwj1OQ6ezhj+aRg2MoF/HQsHx/sOi11JCIiasFYRqhZhHdqg9jH+gIAPtt7HuuScyVORERELRXLCDWbJ4J88dKDnQEAr20+gQNnSyRORERELRHLCDWrucO6YkyAN2oMIl74NhVnCsukjkRERC0Mywg1K0EQ8J8n+iKkfWuU6WowZXUyist4yy8REf2JZYSandJGjhWTgtHBzRGXS29i2lfJuFnFW36JiOgWlhEyidaOCqyaHIJWDrY4ekmDOevSecsvEREBYBkhE+rg5ogVk4KhkMuw82Qh/r0jS+pIRETUArCMkEmFdnDF+0/euuV3RWI2vv39osSJiIhIaiwjZHJjAnwwb1hXAMCiH05i7+kiiRMREZGUWEZIErMe7IzH+/lCbxAxc006MvO1UkciIiKJsIyQJARBQOxjfXBfR1eU627N8luorZQ6FhERSYBlhCSjsJFh+dPB6OjuiHxNJabGJ6NCVyN1LCIiMjGWEZKUysEW8ZND0cZRgZNXtJidkA49b/klIrIqLCMkOb82DlgRFQyFjQy/ZBbh/7adkjoSERGZkNFlJDExEREREfD29oYgCNiyZcsdx+fn52PixIno2rUrZDIZ5syZ08ioZMmC2rXG4nEBAIDVBy8g/mCOtIGIiMhkjC4jFRUV8Pf3R1xcXIPG63Q6uLu7Y8GCBfD39zc6IFmP0X3b4tUR3QEAb/10Cr+cKpQ4ERERmYKNsRuMHDkSI0eObPD49u3bY+nSpQCAVatWGbs7sjLTB3fExasVSEjOw6y16dgwPRy9fVRSxyIiombUIq8Z0el00Gq1dRayDoIg4O2xvXF/ZzfcrNZjanwyrpTelDoWERE1oxZZRmJjY6FSqWoXtVotdSQyIVu5DJ893Q9dPZ1QVKbD1PhklPOWXyIii9Uiy0hMTAw0Gk3tkpeXJ3UkMjEXO1usmhwCNyclsgrKMOO7NNToDVLHIiKiZtAiy4hSqYSLi0udhayPb2sHrIwOhp2tDPvOFGPRDychinwGCRGRpWmRZYToD/7qVlgSGQhBAL47kouVB3jLLxGRpTG6jJSXlyMjIwMZGRkAgJycHGRkZCA3NxfArZ9YoqKi6mzzx/jy8nIUFxcjIyMDp07xwVbUMCN6e+G1UT0AAO9sz8SOEwUSJyIioqYkiEae9967dy+GDBnyt/XR0dGIj4/H5MmTceHCBezdu/fPnQjC38a3a9cOFy5caNA+tVotVCoVNBoNf7KxUqIo4vWtJ/Dt77mws5Vh3XPh8Fe3kjoWERHdQUO/v40uI1JgGSEAqNEbMO3rFOw9XQw3JyU2v9gfalcHqWMREVE9Gvr9zWtGyGzYyGX4dGI/dPdyRkn5rVt+NTerpY5FRET3iGWEzIqT0garp4TA00WJs0XlmPFdGqp5yy8RkVljGSGz01Zlj5XRIXBQyHHgXAkWbD7BW36JiMwYywiZpd4+KnwyIRAyAViXkodl+7KljkRERI3EMkJm66EenlgU0QsA8N6OLPx07IrEiYiIqDFYRsisRfdvjykD2gMA5q0/itSL16UNRERERmMZIbO3YHRPDO3hgaoaA579OgW5V29IHYmIiIzAMkJmTy4TsHR8IHr7uOBaRRUmxydBc4O3/BIRmQuWEbIIjkobrIwOgbfKDtnFFXj+2xRU1fCWXyIic8AyQhbD08UOKyeHwElpg9+zr2H+pmO85ZeIyAywjJBF6dHWBXFP9YNcJmBT2mV88us5qSMREdFdsIyQxRnc1R1vjbl1y+9Hu89gS/pliRMREdGdsIyQRXoqrB2eG9QRAPDKxmNIyrkmcSIiIqoPywhZrPkjumNkby9U6Q147psU5JRUSB2JiIhug2WELJZMJuCjcQHwV7dC6Y1qTFmdhGsVVVLHIiKi/8EyQhbNXiHHl1HB8GlljwtXb+C5r1NQWa2XOhYREf0FywhZPHdnJeKnhMDZzgYpF6/jlY285ZeIqCVhGSGr0MXTGcueDoKNTMAPR69g8e4zUkciIqL/YhkhqzGgsxvefbQPAODjX89hY+oliRMRERHAMkJWZlyIGjOGdAIAxGw6hkPnSyRORERELCNkdf45rBv+0bctqvUipn+TinNFZVJHIiKyaiwjZHVkMgEfPOmPoHatoa2swZT4ZJSU66SORURktVhGyCrZ2cqxYlIQ/FwdkHftJp7lLb9ERJJhGSGr1cZJidVTQqCyt0V6bin+uf4oDAbe8ktEZGosI2TVOrk7YfmkINjKBWw7no//7DwtdSQiIqvDMkJW776ObfDe430BAMv2ncfapFyJExERWReWESIAj/XzxeyHugAAFmw5gf1niyVORERkPVhGiP5rztAueDTQB3qDiBe/TcPpAt7yS0RkCiwjRP8lCAL+/XgfhHZwRZmuBlPjk1FUVil1LCIii8cyQvQXSptbt/x2dHPE5dKbmPZVCm5U1Ugdi4jIohldRhITExEREQFvb28IgoAtW7bcdZu9e/eiX79+UCqV6Ny5M+Lj4xsRlcg0WjkosGpyCFo72OLYJQ1mJ2RAz1t+iYiajdFlpKKiAv7+/oiLi2vQ+JycHIwePRpDhgxBRkYG5syZg2nTpmHnzp1GhyUylfZujvgiKhgKuQy7TxUidnum1JGIiCyWIIpio/+XTxAEbN68GWPHjq13zKuvvopt27bhxIkTtevGjx+P0tJS7Nixo0H70Wq1UKlU0Gg0cHFxaWxcIqP9cPQKXlqbDgB4e0wvTApvL20gIiIz0tDv72a/ZuTw4cMYOnRonXXDhw/H4cOH691Gp9NBq9XWWYik8Ii/N14e3g0AsOiHk/gtq0jiRERElqfZy0hBQQE8PT3rrPP09IRWq8XNmzdvu01sbCxUKlXtolarmzsmUb1efKATngzyhUEEZq5Jw6krLMdERE2pRd5NExMTA41GU7vk5eVJHYmsmCAIeOfRPujfqQ0qqvSYGp+MAg1v+SUiairNXka8vLxQWFhYZ11hYSFcXFxgb29/222USiVcXFzqLERSUtjI8PnTQejs4YQCbSWmxiejQsdbfomImkKzl5Hw8HDs2bOnzrrdu3cjPDy8uXdN1KRU9rZYPTkEbk4KnMrXYtbadN7yS0TUBIwuI+Xl5cjIyEBGRgaAW7fuZmRkIDf31uRiMTExiIqKqh0/ffp0ZGdn45VXXkFWVhY+++wzrF+/HnPnzm2aV0BkQmpXB3wRFQyljQy/ZhXh7Z9OSR2JiMjsGV1GUlJSEBgYiMDAQADAvHnzEBgYiIULFwIA8vPza4sJAHTo0AHbtm3D7t274e/vjw8//BBffvklhg8f3kQvgci0Av1aY0lkAAAg/tAFrDqQI20gIiIzd0/PGTEVPmeEWqLl+84j9ucsCAKw/OkgPNzLS+pIREQtSot5zgiRpXpuUEdMCPWDKAKzEzJw/JJG6khERGaJZYSokQRBwFtjemFgFzfcrNZj6lfJuFx6+2fnEBFR/VhGiO6BrVyGz57qh+5ezigu0+GZ+GSUVVZLHYuIyKywjBDdI2c7W6ycHAJ3ZyWyCsowY006qvUGqWMREZkNlhGiJuDTyh6rokNgbytH4pliLPrhJMzg2nAiohaBZYSoifTxVWHp+AAIArDmSC7+/XMWz5AQETUAywhRE3q4lxdeH90TALA8MRtj4w7idEGZxKmIiFo2lhGiJjb1/g74ZEIgWjnY4uQVLSI+OYDP957no+OJiOrBMkLUDCL8vbFr7iA81N0DVXoD3tuRhSeXHUJ2cbnU0YiIWhyWEaJm4uFshy+jg/H+E33hrLRBWm4pRn28H/EHc2DgWRIiolosI0TNSBAEPBmsxo65g3B/ZzdUVhvwxo+n8NSXR5B37YbU8YiIWgSWESIT8Gllj6+nhuLtMb1gbyvH4eyrGLEkEQlJubwFmIisHssIkYnIZAImhbfHz7MHIqR9a1RU6TF/03FMiU9GgaZS6nhERJJhGSEysfZujkh4LhyvjeoBhY0Me08X4+HF+7Al/TLPkhCRVWIZIZKAXCbg2UEdsf2l+9HXVwVtZQ3mrMvAC9+moaRcJ3U8IiKTYhkhklBnD2dseqE//jmsK2xkAnacLMDwxYnYcSJf6mhERCbDMkIkMRu5DLMe6oKtMwegu5czrlZUYfq3aZiTkA7NDc4ATESWj2WEqIXo5a3C1pkDMGNIJ8gEYEvGFTy8ZB9+O10kdTQiombFMkLUgiht5Hh5eHd8/0J/dHRzRKFWhymrkzH/+2Moq+RZEiKyTCwjRC1QoF9rbHtpIKYO6ABBABKS8zBiyX4cOl8idTQioibHMkLUQtkr5FgY0RNrn70Pald7XC69iYlfHMEbP5zEzSq91PGIiJoMywhRC3dfxzb4efYgTAzzAwDEH7qAUR/vR+rF6xInIyJqGiwjRGbASWmDdx/tg6+mhsLLxQ45JRV4ctkh/PvnLOhqeJaEiMwbywiRGRnc1R075w7CY/18YBCBZfvO45FPDuLEZY3U0YiIGo1lhMjMqOxt8dG4ACyfFAQ3JwVOF5ZhbNxBLPnlDKr1BqnjEREZjWWEyEwN7+WFnXMGYVQfL9QYRCz55Swe/ewgzhSWSR2NiMgoLCNEZqyNkxJxE/vh4wmBUNnb4sRlLf7x8QEs33ceegMn3SMi88AyQmTmBEHAI/7e2D13EB7s7oEqvQGxP2dh3PLDyCmpkDoeEdFdsYwQWQgPFzusjA7Gf57oCyelDVIvXsfIpYn46tAFGHiWhIhaMJYRIgsiCALGBauxY85A9O/UBpXVBiz64SSeXnkEl67fkDoeEdFtNaqMxMXFoX379rCzs0NYWBiSkpLqHVtdXY233noLnTp1gp2dHfz9/bFjx45GByaiu/Nt7YBvnwnDW2N6wd5WjkPnr2LEkv1Yl5wLUeRZEiJqWYwuI+vWrcO8efOwaNEipKWlwd/fH8OHD0dR0e1nFl2wYAGWL1+OTz75BKdOncL06dPx6KOPIj09/Z7DE1H9ZDIBUeHt8fPsgQhu1xrluhq8+v1xTI1PRqG2Uup4RES1BNHI/00KCwtDSEgIPv30UwCAwWCAWq3GrFmzMH/+/L+N9/b2xmuvvYYZM2bUrnv88cdhb2+Pb7/9tkH71Gq1UKlU0Gg0cHFxMSYuEQHQG0SsPJCND3adQVWNASp7W7w1phce8feGIAhSxyMiC9XQ72+jzoxUVVUhNTUVQ4cO/fMfkMkwdOhQHD58+Lbb6HQ62NnZ1Vlnb2+PAwcO1LsfnU4HrVZbZyGixpPLBDw3qBO2zboffXxU0NysxuyEDLz4XRquluukjkdEVs6oMlJSUgK9Xg9PT8866z09PVFQUHDbbYYPH46PPvoIZ8+ehcFgwO7du7Fp0ybk5+fXu5/Y2FioVKraRa1WGxOTiOrRxdMZm17sj3nDusJGJuDnEwV4eHEidp68/X+/RESm0Ox30yxduhRdunRB9+7doVAoMHPmTEyZMgUyWf27jomJgUajqV3y8vKaOyaR1bCVy/DSQ12wZcYAdPN0xtWKKjz/TSrmrsuA5ka11PGIyAoZVUbc3Nwgl8tRWFhYZ31hYSG8vLxuu427uzu2bNmCiooKXLx4EVlZWXByckLHjh3r3Y9SqYSLi0udhYiaVm8fFX6YNQAvPNAJMgHYnH4ZDy/Zh72nb38xOhFRczGqjCgUCgQFBWHPnj216wwGA/bs2YPw8PA7bmtnZwcfHx/U1NTg+++/x5gxYxqXmIiajNJGjldHdMfGF/qjo5sjCrU6TF6djJhNx1Guq5E6HhFZCaN/ppk3bx6++OILfPXVV8jMzMQLL7yAiooKTJkyBQAQFRWFmJiY2vFHjhzBpk2bkJ2djf3792PEiBEwGAx45ZVXmu5VENE96efXGtteGogpA9oDANYm5WLEkkQcPn9V2mBEZBVsjN0gMjISxcXFWLhwIQoKChAQEIAdO3bUXtSam5tb53qQyspKLFiwANnZ2XBycsKoUaPwzTffoFWrVk32Iojo3tkr5FgU0QsP9/TCyxuP4tL1m5jwxe+YMqA9XhneHfYKudQRichCGf2cESnwOSNEplWuq8E72zKxNikXANDRzREfjPNHP7/WEicjInPSLM8ZISLr4KS0QexjfRA/JQSeLkpkl1Tgic8P4b0dWdDV6KWOR0QWhmWEiOr1QDcP7JozGI8G+sAgAp/vPY9HPjmIE5c1UkcjIgvCMkJEd6RysMXiyAAsezoIbRwVOF1YhrFxB7H0l7Oo1hukjkdEFoBlhIgaZERvL+yaOwgje3uhxiBi8S9n8Nhnh3C2sEzqaERk5lhGiKjB2jgp8dlT/bB0fABU9rY4flmD0Z8cwIrE89AbWvy18ETUQrGMEJFRBEHAmAAf7Jo7CEO6uaOqxoB3t2chcvlhXCipkDoeEZkhlhEiahRPFzusmhyC9x7vAyelDVIuXsfIpfvx9eELMPAsCREZgWWEiBpNEAREhvhhx5yBCO/YBjer9Vi49SQmrTqCy6U3pY5HRGaCZYSI7plvawd8Ny0Mbz7SC3a2Mhw8dxXDFydifXIezOC5ikQkMZYRImoSMpmA6P7t8fPsQQhq1xrluhq88v0xPPNVCoq0lVLHI6IWjGWEiJpUBzdHrH8+HDEju0Mhl+HXrCIMW5yIH45e4VkSIrotlhEianJymYDnB3fCTy/dj94+LtDcrMZLa9MxY00arpbrpI5HRC0MywgRNZuuns7Y/OIAzBnaBTYyAduPF2D4kkTsPFkgdTQiakFYRoioWdnKZZgztCu2zBiArp5OKCmvwvPfpGLeugxoblZLHY+IWgCWESIyid4+Kvw4635MH9wJMgHYlH4ZwxcnYt+ZYqmjEZHEWEaIyGSUNnLMH9kdG6b3Rwc3RxRoKxG9Kgn/2nwc5boaqeMRkURYRojI5ILatcb2lwZicv/2AIA1R3Ixcmkifs++Km0wIpIEywgRScJeIccbj/TCmmfD4NPKHnnXbmLCF7/jrR9PobJaL3U8IjIhlhEiklT/Tm7YMWcgJoSqIYrAqoM5GPXxfqTnXpc6GhGZCMsIEUnO2c4WsY/1xeopIfBwViK7uAKPf34I/9mRBV0Nz5IQWTqWESJqMYZ088CuuYMwNsAbBhH4bO95jPn0IE5e0UgdjYiaEcsIEbUorRwUWDI+EMue7oc2jgpkFZRhzKcH8cmes6jRG6SOR0TNgGWEiFqkEb3bYufcQRjRyws1BhEf7j6Dxz8/hHNFZVJHI6ImxjJCRC2Wm5MSnz/dD0siA+BiZ4OjlzQY9fEBfJGYDb2Bk+4RWQqWESJq0QRBwNhAH+yaOxgPdHNHVY0B72zPROTyw7hQUiF1PCJqAiwjRGQWvFR2WD05BP9+rA8cFXKkXLyOkUv345vDF2DgWRIis8YyQkRmQxAEjA/1w445g3BfR1fcrNbj9a0nEbUqCZdLb0odj4gaiWWEiMyO2tUBa6bdh0URPWFnK8OBcyUYsTgRG1LyIIo8S0JkblhGiMgsyWQCpgzogO0vDUSgXyuU6Wrw8sZjePbrFBSVVUodj4iMwDJCRGato7sTNk7vj/kju0Mhl+GXzCI8vDgRPx69InU0ImqgRpWRuLg4tG/fHnZ2dggLC0NSUtIdxy9ZsgTdunWDvb091Go15s6di8pK/p8LETUNuUzA9MGd8OOs+9HL2wWlN6oxa206ZnyXhmsVVVLHI6K7MLqMrFu3DvPmzcOiRYuQlpYGf39/DB8+HEVFRbcdv2bNGsyfPx+LFi1CZmYmVq5ciXXr1uFf//rXPYcnIvqrbl7O2DJjAGY/1AVymYBtx/Px8OJ92H2qUOpoRHQHgmjk1V5hYWEICQnBp59+CgAwGAxQq9WYNWsW5s+f/7fxM2fORGZmJvbs2VO77p///CeOHDmCAwcONGifWq0WKpUKGo0GLi4uxsQlIit1/JIG89Zn4GxROQDg8X6+WBjREyp7W4mTEVmPhn5/G3VmpKqqCqmpqRg6dOif/4BMhqFDh+Lw4cO33aZ///5ITU2t/SknOzsb27dvx6hRo+rdj06ng1arrbMQERmjj68KP866H88P7ghBAL5Pu4QRSxKReKZY6mhE9D+MKiMlJSXQ6/Xw9PSss97T0xMFBQW33WbixIl46623cP/998PW1hadOnXCAw88cMefaWJjY6FSqWoXtVptTEwiIgCAna0cMSN7YOP0cLRv44B8TSWiViXhtc3HUaGrkToeEf1Xs99Ns3fvXrz77rv47LPPkJaWhk2bNmHbtm14++23690mJiYGGo2mdsnLy2vumERkwYLauWL77IGY3L89AOC7I7kYuXQ/jmRflTYYEQEAbIwZ7ObmBrlcjsLCuheDFRYWwsvL67bbvP7665g0aRKmTZsGAOjTpw8qKirw3HPP4bXXXoNM9vc+pFQqoVQqjYlGRHRHDgobvPFILzzc0xMvbzyG3Gs3MP6L3zF1QAe8PLwb7GzlUkckslpGnRlRKBQICgqqczGqwWDAnj17EB4eftttbty48bfCIZff+o+eT0okIlPr39kNO+YMRGSwGqIIrDyQg1Ef70dGXqnU0YisltE/08ybNw9ffPEFvvrqK2RmZuKFF15ARUUFpkyZAgCIiopCTExM7fiIiAh8/vnnSEhIQE5ODnbv3o3XX38dERERtaWEiMiUnO1s8d4TfbFqcjA8nJXILq7AY58dxAc7T6OqxiB1PCKrY9TPNAAQGRmJ4uJiLFy4EAUFBQgICMCOHTtqL2rNzc2tcyZkwYIFEAQBCxYswOXLl+Hu7o6IiAi88847TfcqiIga4cHuntg1tzUW/XASWzOu4NPfzuGXzEJ8NC4APb35GAEiUzH6OSNS4HNGiKi5/Xw8H69tOYFrFVWwlQuY/VAXTB/cCTZyzppB1FjN8pwRIiJLNbJPW+yaOwgP9/REtV7EB7vO4PHPD+FcUZnU0YgsHssIEdF/uTkpsXxSEBZH+sPZzgZHL2kw6uMD+HJ/NvSGFn8SmchssYwQEf2FIAh4NNAXu+YOwqCu7qiqMeD/tmViworfcfFqhdTxiCwSywgR0W20VdnjqykhiH2sDxwVciRduIaRS/fj298v8rEERE2MZYSIqB6CIGBCqB92zBmEsA6uuFGlx4ItJxC1KglXSm9KHY/IYrCMEBHdhdrVAWufvQ+LInpCaSPD/rMlGL44ERtTL/EsCVETYBkhImoAmUzAlAEdsH32QAT6tUKZrgb/b8NRPPt1CorKKqWOR2TWWEaIiIzQyd0JG54PxysjusFWLuCXzCI8vDgRPx27InU0IrPFMkJEZCQbuQwvPtAZP866Hz3buqD0RjVmrknHzDVpuF5RJXU8IrPDMkJE1EjdvVywZcYAvPRQF8hlAn46lo9hixPxy6nCu29MRLVYRoiI7oHCRoZ5w7pi84v90dnDCSXlOkz7OgX/b8NRaCurpY5HZBZYRoiImkBf31b4adb9eH5QRwgCsDH1EkYsTsSBsyVSRyNq8VhGiIiaiJ2tHDGjemDD8+Fo18YBVzSVeHrlESzYchwVuhqp4xG1WCwjRERNLLi9K36ePRBR4e0AAN/+nouRS/cjKeeaxMmIWiaWESKiZuCgsMFbY3rju2lh8FbZIffaDUSuOIx3tp1CZbVe6nhELQrLCBFRMxrQ2Q075g7CuGBfiCLwxf4cjP54P47mlUodjajFYBkhImpmLna2+M8T/lgZHQx3ZyXOF1fgsc8P4cNdp1FVY5A6HpHkWEaIiEzkoR6e2DVnEB7x94beIOKTX89hTNxBZOZrpY5GJCmWESIiE2rtqMDHEwIRN7EfWjvYIjNfi0c+PYC4386hRs+zJGSdWEaIiCQwum9b7Jo7GMN6eqJaL+L9nafx+LLDOFdULnU0IpNjGSEikoi7sxIrJgXho3H+cLazwdG8Uoz+eD++3J8Ng0GUOh6RybCMEBFJSBAEPNbPF7vmDsLALm7Q1Rjwf9syMf6L35F79YbU8YhMgmWEiKgFaKuyx9dTQ/HOo73hoJAjKecaRixNxHdHLkIUeZaELBvLCBFRCyEIAp4Ka4cdswchtIMrblTp8drmE4halYR8zU2p4xE1G5YRIqIWxq+NAxKevQ+v/6MnlDYy7D9bgocXJ+L71Es8S0IWiWWEiKgFkskEPHN/B2yfPRAB6lYoq6zBPzccRdSqJPxyqpC3AZNFEUQzqNlarRYqlQoajQYuLi5SxyEiMqkavQHLE7Ox5JczqNbf+sj2crHDk8G+GBeshtrVQeKERLfX0O9vlhEiIjORU1KB736/iO/TLuH6jWoAgCAA93d2w4RQPwzt4QmFDU94U8vBMkJEZKF0NXrsOlmIhORcHDx3tXZ9G0cFngjyRWSIGh3dnSRMSHQLywgRkRW4eLUC65LzsCH1EorLdLXrwzq4YkKoH0b09oKdrVzChGTNGvr93ajzeXFxcWjfvj3s7OwQFhaGpKSkesc+8MADEAThb8vo0aMbs2siIvqLdm0c8cqI7jg0/0GsmBSEB7t7QCYAR3KuYc66DIS9uwdv/HASWQWcjI9aLqPPjKxbtw5RUVFYtmwZwsLCsGTJEmzYsAGnT5+Gh4fH38Zfu3YNVVVVtX++evUq/P398eWXX2Ly5MkN2ifPjBARNdyV0pvYkHIJ61PycLn0z+eTBKhbYUKoGv/o6w1HpY2ECclaNNvPNGFhYQgJCcGnn34KADAYDFCr1Zg1axbmz59/1+2XLFmChQsXIj8/H46Ojg3aJ8sIEZHx9AYR+88WIyEpD79kFqLmv/PdOCrkeCTABxNC1ejjo4IgCBInJUvVLGWkqqoKDg4O2LhxI8aOHVu7Pjo6GqWlpdi6detd/40+ffogPDwcK1asaOhuWUaIiO5RcZkOG1MvYV1yLi78Zc6bnm1dMCFUjUcCfKCyt5UwIVmihn5/G3WerqSkBHq9Hp6ennXWe3p6Iisr667bJyUl4cSJE1i5cuUdx+l0Ouh0f16IpdXyt04ionvh7qzECw90wvTBHXE4+yrWJefh5xMFOJWvxetbT+Kd7ZkY1actJoT6Ibhda54tIZMy6Y+GK1euRJ8+fRAaGnrHcbGxsXjzzTdNlIqIyHoIgoD+ndzQv5Mb3qiowub0y0hIzsWZwnJsSruMTWmX0dnDCeND1Hisny9cHRVSRyYrYLKfaSoqKuDt7Y233noLs2fPvuN+bndmRK1W82caIqJmIIoi0nJLkZCUi5+O5eNmtR4AoJDL8HAvT0wI9UN4xzaQyXi2hIzTLD/TKBQKBAUFYc+ePbVlxGAwYM+ePZg5c+Ydt92wYQN0Oh2efvrpu+5HqVRCqVQaE42IiBpJEAQEtWuNoHat8XpET/yQcQUJybk4cVmLn47l46dj+fBzdUBkiBpPBvnCw8VO6shkYRp1a290dDSWL1+O0NBQLFmyBOvXr0dWVhY8PT0RFRUFHx8fxMbG1tlu4MCB8PHxQUJCgtEheQErEZHpnbiswdqkXGzNuIJyXQ0AQC4T8FB3D4wPVWNwVw/IebaE7qBZzowAQGRkJIqLi7Fw4UIUFBQgICAAO3bsqL2oNTc3FzJZ3WepnT59GgcOHMCuXbuM3R0REUmkt48K7zzaB6+N7oFtx/KRkJyH1IvXsetUIXadKkRblR2eDFYjMkQNn1b2UsclM8bHwRMRUYOdKSxDQlIeNqVfQulfJusb1MUdE0LVeKiHJ2zlnKyPbuHcNERE1Gwqq/XYebIACUl5OJz952R9bk5KPBHki/EharR3a9iDLclysYwQEZFJXCipQEJyHjamXkJJ+Z93QoZ3bIPxoWoM78XJ+qwVywgREZlUtd6APZlFSEjOxb4zxfjj26WVgy0eDfTBhFA/dPV0ljYkmRTLCBERSeZy6U2sT87D+pQ85Gsqa9f382uF8aF++EfftnBQcLI+S8cyQkREktMbRCSeKcbapFzsySqC/r+T9TkrbfBIgDcmhPqht49K4pTUXFhGiIioRSnSVmJD6iWsS85D7rU/J+vr7eOC8SF+GBPgDWc7TtZnSVhGiIioRTIYRPyefRVrk/Ow80QBqvQGAIC9rRz/6NsW40PV6OfHyfosAcsIERG1eNcqqrAp7RISkvNwrqi8dn1XTydEhvjhsUAftOZkfWaLZYSIiMyGKIpIvXgda5PysO34FVRW3zpborCRYUQvL4wPVSO8YxueLTEzLCNERGSWNDer8UPGZaxNysOpfG3t+vZtHBAZ4ocngnzh7szJVM0BywgREZk1URRx/LIGCcl5+OEvk/XZyAQM7eGJyFA1BnVx52R9LRjLCBERWYwKXQ22HcvH2uRcpOeW1q73aWWPJ4N9MS5YDW9O1tfisIwQEZFFyirQIiEpD5vTL0Nz89ZkfTIBGNzVHeND/fBgdw9O1tdCsIwQEZFFq6zWY8eJAqxNysWRnGu1692dlXgyyBeRIWq0a8PJ+qTEMkJERFYju7gc61Ly8H3qJZSUV9WuH9C5DSJD/DC8lyeUNpysz9RYRoiIyOpU1RiwJ7MQa5PzsP/sn5P1tXawxWP9fDEhVI3OHpysz1RYRoiIyKrlXbuBDSl5WJ9yCQXaPyfrC27XGuND/TC6T1vYK3i2pDmxjBAREQGo0Ruw70wx1ibl4bfTf5msz84GYwN8MD5UjV7enKyvObCMEBER/Y9CbSU2pl5CQnIu8q7drF3f11eF8SF+eCTAG05KGwkTWhaWESIionoYDCIOnb+Ktcm52HWyANX6W1+FDgo5Ivp6IzJUjUB1Kz5+/h6xjBARETXA1XIdNqVdxtrkXGQXV9Su7+bpjPGhajwa6INWDpysrzFYRoiIiIwgiiKSL1xHQlIuth3Ph67mz8n6RvX2wvhQP4R1cOXZEiOwjBARETWS5kY1th69NVlf5l8m6+vo5ojIEDUeD/KFmxMn67sblhEiIqJ7JIoijl3SICE5Fz9kXEFFlR4AYCsXMKynJyJD/DCwsxtknKzvtlhGiIiImlC5rgY/Hb2Ctcl5OJpXWrvep5U9IkPUGBeshpfKTrqALRDLCBERUTPJzNciISkXm9MvQ1tZA+DWZH1DunlgfKgfhnRzhw0n62MZISIiam6V1Xr8fCIfa5PykPSXyfo8XZR4MkiNyBA11K4OEiaUFssIERGRCZ0vLse65DxsTL2EaxV/TtY3sIsbxof4YVhPTyhsrOtsCcsIERGRBKpqDNh9qhAJybnYf7akdr2rowKP9/PB+FA/dHJ3kjCh6bCMEBERSSzv2g2sS87DhtQ8FGp1tetD27tifKgao/q0hZ2t5U7W19Dv70adL4qLi0P79u1hZ2eHsLAwJCUl3XF8aWkpZsyYgbZt20KpVKJr167Yvn17Y3ZNRERkNtSuDvh/w7vh4KsP4ouoYAzt4QGZACRduIZ5648i9J1fsGjriTrPMrFGRp8ZWbduHaKiorBs2TKEhYVhyZIl2LBhA06fPg0PD4+/ja+qqsKAAQPg4eGBf/3rX/Dx8cHFixfRqlUr+Pv7N2ifPDNCRESWokBTiQ0peUhIzsPl0j8n6/NXt8KEEDUi/L3haCGT9TXbzzRhYWEICQnBp59+CgAwGAxQq9WYNWsW5s+f/7fxy5Ytw/vvv4+srCzY2toa+TJuYRkhIiJLYzCIOHCuBAnJudh9qrB2sj5HhRwR/t4YH+oHf1+VWT9+vlnKSFVVFRwcHLBx40aMHTu2dn10dDRKS0uxdevWv20zatQouLq6wsHBAVu3boW7uzsmTpyIV199FXJ5w34nYxkhIiJLVlKuw/epl7AuOQ/ZJX9O1tfdyxkTQv0wNtAHKvvG/Q+9lBr6/W3UeaCSkhLo9Xp4enrWWe/p6YmsrKzbbpOdnY1ff/0VTz31FLZv345z587hxRdfRHV1NRYtWnTbbXQ6HXS6Py/00Wqt+7c0IiKybG5OSjw/uBOeG9QRR3KuYV1yHrYdz0dWQRkW/XAS727PxOg+bTE+1A8h7Vub9dmS22n2H6UMBgM8PDywYsUKyOVyBAUF4fLly3j//ffrLSOxsbF48803mzsaERFRiyIIAu7r2Ab3dWyDNyJ6YXP6JSQk5yGroAyb0i9jU/pldHJ3xPgQPzzWzwdtLGSyPqPupnFzc4NcLkdhYWGd9YWFhfDy8rrtNm3btkXXrl3r/CTTo0cPFBQUoKqq6rbbxMTEQKPR1C55eXnGxCQiIjJ7KgdbTB7QAT/PHojNL/ZHZLAaDgo5zhdX4J3tmbgvdg9mrEnDgbMlMBha/FM67sioMqJQKBAUFIQ9e/bUrjMYDNizZw/Cw8Nvu82AAQNw7tw5GAyG2nVnzpxB27ZtoVAobruNUqmEi4tLnYWIiMgaCYKAQL/WeO+Jvkh6bSjefbQP+vqqUK0Xse1YPp5eeQSDP/gNn/56FoXaSqnjNkqjbu2Njo7G8uXLERoaiiVLlmD9+vXIysqCp6cnoqKi4OPjg9jYWABAXl4eevXqhejoaMyaNQtnz57F1KlT8dJLL+G1115r0D55ASsREVFdJ69osC45D5vTL6Psv5P1yWUChnTzwIRQNQZ3lX6yvma5gBUAIiMjUVxcjIULF6KgoAABAQHYsWNH7UWtubm5kMn+fPFqtRo7d+7E3Llz0bdvX/j4+GD27Nl49dVXG/GyiIiICAB6eavw1hgVYkb2wPbj+UhIzkXyhev4JbMQv2QWwsvFDuOCfTEuRA3f1i17sj4+Dp6IiMhCnCsqQ0JSHr5Pu4TrN6oBAIIADOzijgkhajzUw7ST9XFuGiIiIiulq9Fj18lbk/UdPHe1dr2bkwKP9/NFZIgaHU0wWR/LCBEREeHi1QqsT8nDhpRLKCr78xleYR1cMSHUDyN6ezXbZH0sI0RERFSrRm/Ar1lFSEjOw97TRfjjbmCVvS0eDfTB0/e1Q2ePpj1b0mwXsBIREZH5sZHL8HAvLzzcywtXSm9iQ8olrE+5NVlf/KEL6OTh1ORlpMHZJNkrERERSca7lT1mD+2CmQ92xv6zxdiYegmP+HtLlodlhIiIyErJZQIe6OaBB7p5SJpD2qehEBERkdVjGSEiIiJJsYwQERGRpFhGiIiISFIsI0RERCQplhEiIiKSFMsIERERSYplhIiIiCTFMkJERESSYhkhIiIiSbGMEBERkaRYRoiIiEhSLCNEREQkKbOYtVcURQCAVquVOAkRERE11B/f2398j9fHLMpIWVkZAECtVkuchIiIiIxVVlYGlUpV798L4t3qSgtgMBhw5coVODs7QxCEJvt3tVot1Go18vLy4OLi0mT/rqXi8Wo4HquG47FqOB6rhuOxarjmPFaiKKKsrAze3t6Qyeq/MsQszozIZDL4+vo227/v4uLCN6sReLwajseq4XisGo7HquF4rBquuY7Vnc6I/IEXsBIREZGkWEaIiIhIUlZdRpRKJRYtWgSlUil1FLPA49VwPFYNx2PVcDxWDcdj1XAt4ViZxQWsREREZLms+swIERERSY9lhIiIiCTFMkJERESSYhkhIiIiSVl8GYmLi0P79u1hZ2eHsLAwJCUl3XH8hg0b0L17d9jZ2aFPnz7Yvn27iZJKz5hjFR8fD0EQ6ix2dnYmTCudxMREREREwNvbG4IgYMuWLXfdZu/evejXrx+USiU6d+6M+Pj4Zs/ZEhh7rPbu3fu395UgCCgoKDBNYAnFxsYiJCQEzs7O8PDwwNixY3H69Om7bmeNn1mNOVbW+pn1+eefo2/fvrUPNAsPD8fPP/98x22keE9ZdBlZt24d5s2bh0WLFiEtLQ3+/v4YPnw4ioqKbjv+0KFDmDBhAp555hmkp6dj7NixGDt2LE6cOGHi5KZn7LECbj2tLz8/v3a5ePGiCRNLp6KiAv7+/oiLi2vQ+JycHIwePRpDhgxBRkYG5syZg2nTpmHnzp3NnFR6xh6rP5w+fbrOe8vDw6OZErYc+/btw4wZM/D7779j9+7dqK6uxsMPP4yKiop6t7HWz6zGHCvAOj+zfH198e9//xupqalISUnBgw8+iDFjxuDkyZO3HS/Ze0q0YKGhoeKMGTNq/6zX60Vvb28xNjb2tuPHjRsnjh49us66sLAw8fnnn2/WnC2Bscdq9erVokqlMlG6lguAuHnz5juOeeWVV8RevXrVWRcZGSkOHz68GZO1PA05Vr/99psIQLx+/bpJMrVkRUVFIgBx37599Y6x5s+sv2rIseJn1p9at24tfvnll7f9O6neUxZ7ZqSqqgqpqakYOnRo7TqZTIahQ4fi8OHDt93m8OHDdcYDwPDhw+sdbykac6wAoLy8HO3atYNarb5j07Z21vq+uhcBAQFo27Ythg0bhoMHD0odRxIajQYA4OrqWu8YvrduacixAviZpdfrkZCQgIqKCoSHh992jFTvKYstIyUlJdDr9fD09Kyz3tPTs97fnwsKCowabykac6y6deuGVatWYevWrfj2229hMBjQv39/XLp0yRSRzUp97yutVoubN29KlKplatu2LZYtW4bvv/8e33//PdRqNR544AGkpaVJHc2kDAYD5syZgwEDBqB37971jrPWz6y/auixsubPrOPHj8PJyQlKpRLTp0/H5s2b0bNnz9uOleo9ZRaz9lLLEx4eXqdZ9+/fHz169MDy5cvx9ttvS5iMzFm3bt3QrVu32j/3798f58+fx+LFi/HNN99ImMy0ZsyYgRMnTuDAgQNSR2nxGnqsrPkzq1u3bsjIyIBGo8HGjRsRHR2Nffv21VtIpGCxZ0bc3Nwgl8tRWFhYZ31hYSG8vLxuu42Xl5dR4y1FY47V/7K1tUVgYCDOnTvXHBHNWn3vKxcXF9jb20uUynyEhoZa1ftq5syZ+Omnn/Dbb7/B19f3jmOt9TPrD8Ycq/9lTZ9ZCoUCnTt3RlBQEGJjY+Hv74+lS5fedqxU7ymLLSMKhQJBQUHYs2dP7TqDwYA9e/bU+1tZeHh4nfEAsHv37nrHW4rGHKv/pdfrcfz4cbRt27a5Ypota31fNZWMjAyreF+JooiZM2di8+bN+PXXX9GhQ4e7bmOt763GHKv/Zc2fWQaDATqd7rZ/J9l7qlkvj5VYQkKCqFQqxfj4ePHUqVPic889J7Zq1UosKCgQRVEUJ02aJM6fP792/MGDB0UbGxvxgw8+EDMzM8VFixaJtra24vHjx6V6CSZj7LF68803xZ07d4rnz58XU1NTxfHjx4t2dnbiyZMnpXoJJlNWViamp6eL6enpIgDxo48+EtPT08WLFy+KoiiK8+fPFydNmlQ7Pjs7W3RwcBBffvllMTMzU4yLixPlcrm4Y8cOqV6CyRh7rBYvXixu2bJFPHv2rHj8+HFx9uzZokwmE3/55RepXoLJvPDCC6JKpRL37t0r5ufn1y43btyoHcPPrFsac6ys9TNr/vz54r59+8ScnBzx2LFj4vz580VBEMRdu3aJothy3lMWXUZEURQ/+eQT0c/PT1QoFGJoaKj4+++/1/7d4MGDxejo6Drj169fL3bt2lVUKBRir169xG3btpk4sXSMOVZz5sypHevp6SmOGjVKTEtLkyC16f1x++n/Ln8cn+joaHHw4MF/2yYgIEBUKBRix44dxdWrV5s8txSMPVbvvfee2KlTJ9HOzk50dXUVH3jgAfHXX3+VJryJ3e44AajzXuFn1i2NOVbW+pk1depUsV27dqJCoRDd3d3Fhx56qLaIiGLLeU8JoiiKzXvuhYiIiKh+FnvNCBEREZkHlhEiIiKSFMsIERERSYplhIiIiCTFMkJERESSYhkhIiIiSbGMEBERkaRYRoiIiEhSLCNEREQkKZYRIiIikhTLCBEREUmKZYSIiIgk9f8BlMpIDnoZf3YAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "class CustomCallback(keras.callbacks.Callback):\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        keys = list(logs.keys())\n",
        "        if(epoch % EVAL_STEPS == EVAL_STEPS-1):\n",
        "          # Evaluate\n",
        "          display(HTML(inference_test(test_image)))\n",
        "          make_predictions()\n",
        "\n",
        "#history = pali_gemma_lm.fit(train_data, epochs=TRAIN_STEPS, callbacks=[CustomCallback()])\n",
        "history = pali_gemma_lm.fit(train_data, epochs=TRAIN_STEPS)\n",
        "plt.plot(history.history['loss'])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yd9c2gkTJ1px"
      },
      "source": [
        "## Output\n",
        "\n",
        "The validation data for this notebook consists of just 10 images. In normal code, you would likely have many more data points for validation, but for this notebook, run the following code to generate descriptions for all 10 images. After tuning the model, these descriptions should be very similar in form and content coverage to the descriptions included with the training data that you looked at earlier in this notebook.\n",
        "\n",
        "Run the below code to generate descriptions for the validation data set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_VeH5WDFfziX"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A brown cow stands on the shore of a beach, its eyes open and its mouth closed. The cow has a white spot on its head, and its ears are white. The beach is sandy and white, and the sand is white. The sky is clear blue, and the sun is shining.</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Inference Result\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A white clothes rack with white hangers is set up in a room. The rack is on a metal pole and has white and black clothes on it. The clothes on the rack are white and black.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A white hoodie on a hanger, showcasing a long sleeve and a drawstring at the waist. The hoodie has a black string on the front and a black string on the back.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A woman wearing a pink bag and a pair of jeans. The bag is pink and has a black strap. The woman has a gold bracelet on her wrist.</p>\n",
              "    </div>\n",
              "    \n",
              "    <div style=\"display: inline-flex; align-items: center; justify-content: center;\">\n",
              "        <img style=\"width:128px; height:128px;\" src=\"\" />\n",
              "        <p style=\"width:256px; margin:10px; font-size:small;\">caption en\n",
              "A woman holds a black handbag with a gold chain. The handbag has a black and gold chain and a black and silver lock. The woman has skinny blue jeans and black boots.</p>\n",
              "    </div>\n",
              "    "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display(HTML(inference_test(test_image)))\n",
        "make_predictions()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "[PaliGemma_1]Finetune_with_Keras.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
